Papers with pretraining approaches
C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References (2022.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to pretrain open-domain question answering systems lack task-specific annotations. |
| Approach: | They propose to pretrain a two-stage open-domain question answering system with strong transfer capabilities by using a dictionary and a large-scale corpus. |
| Outcome: | The proposed approach leads to 2%-10% gains in top-20 accuracy and improves with reader. |
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs (2025.emnlp-main)
Copied to clipboard
Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Yuanxing Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Wentao Zhang, Feng Zhao
| Challenge: | Multimodal Large Language Models (MLLMs) integrate visual and textual inputs, yet modality alignment remains one of the most challenging aspects. |
| Approach: | They propose a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. |
| Outcome: | The proposed method improves performance across various model sizes, with smaller models benefiting the most. |
Diverse Pretrained Context Encodings Improve Document Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for sentence-level sequence-to-sequence translations do not use extra-sentential information. |
| Approach: | They propose a sentence-level sequence-to-sequence transformer with multiple pre-trained context signals. |
| Outcome: | The proposed model outperforms existing models on Chinese-English and English-German tasks. |
Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. |
| Approach: | They propose two approaches to contextualise visual entities in a multimodal setup by using verbalised scene graphs and masked relation prediction. |
| Outcome: | The proposed models can learn better representations from weakly-supervised relations data. |
Multi-pretraining for Large-scale Text Classification (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for large-scale text classification involve excessive computation and memory overheads. |
| Approach: | They propose a self-supervised and weakly supervised pretraining frameworks for large-scale text classification with multiple categories. |
| Outcome: | The proposed framework improves on the self-supervised and weakly supervised methods while being computationally efficient. |